1 當前存儲信息 以下ASM磁盤組、OCR、VOTE信息: ASM磁盤組: ASMCMD> lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED NORMAL N 512 4096 1048576 3071982 3071091 298 1535396 0 N BACK/ MOUNTED NORMAL N 512 4096 1048576 4095976 1561759 633568 464095 0 N DATA/ MOUNTED NORMAL N 512 4096 1048576 102396 101470 326 50572 0 N OCR/
OCR&VOTE信息: [grid@oracle1 bin]$ ./ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2720 Available space (kbytes) : 259400 ID : 2006438789 Device/File Name : +OCR Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check bypassed due to non-privileged user 由于OCR和VOTEDISK都和ASM放在同一個存儲上,所以OCR和VOTEDISK也需要遷移到新的存儲上。 2 新存儲磁盤分區 要求:(由存儲工程師操作) 2.1. 共享存儲,兩臺服務器都可以看到新存儲分配的磁盤空間。 2.2. 如之前ASM磁盤組所有存儲分區大小,個數保持一致。
3 劃分后磁盤分區 [root@oracle1 sbin]# fdisk -l
Disk /dev/cciss/c0d0: 1000.1 GB, 1000171331584 bytes 255 heads, 63 sectors/track, 121597 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/cciss/c0d0p1 * 1 13 104391 83 Linux /dev/cciss/c0d0p2 14 121597 976623480 8e Linux LVM
Disk /dev/sda: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sda1 1 130541 1048570551 83 Linux
Disk /dev/sdb: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdb1 1 130541 1048570551 83 Linux
Disk /dev/sdc: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdc1 1 130541 1048570551 83 Linux
Disk /dev/sdd: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdd1 1 130541 1048570551 83 Linux
Disk /dev/sde: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sde1 1 130541 1048570551 83 Linux
Disk /dev/sdf: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdf1 1 130541 1048570551 83 Linux
Disk /dev/sdg: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdg1 1 130541 1048570551 83 Linux
Disk /dev/sdh: 107.3 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdh2 1 4000 32129968+ 83 Linux /dev/sdh3 4001 8000 32130000 83 Linux /dev/sdh4 8001 13054 40596255 83 Linux
WARNING: The size of this disk is2.9 TB (2919504019456 bytes). DOS partition table format can not be used on drives for volumes larger than 2.2 TB (2199023255040 bytes). Use parted(1) and GUID partition table format (GPT).
Disk /dev/sdi: 2919.5 GB, 2919504019456 bytes 255 heads, 63 sectors/track, 354942 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdi1 1 130000 1044224968+ 83 Linux /dev/sdi2 130001 267349 1103255842+ 83 Linux
Disk /dev/sdj: 1073.7 GB, 1073741824000 bytes --------------------------------新加盤 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdj doesn't contain a valid partition table
Disk /dev/sdk: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdk doesn't contain a valid partition table
Disk /dev/sdl: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdl doesn't contain a valid partition table
Disk /dev/sdm: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdm doesn't contain a valid partition table
Disk /dev/sdn: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdn doesn't contain a validpartition table
Disk /dev/sdo: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdo doesn't conain a valid partition table
Disk /dev/sdp: 1073.7 GB, 1073741824000 bytes 255 heads, 63 sectors/track, 130541 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdp doesn't contain a valid partition table
Disk /dev/sdq: 107.3 GB, 107374182400 bytes 255 heads, 63 sectors/track, 13054 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdq doesn't contain a valid partition table
Disk /dev/sdr: 2919.5 GB, 2919504019456 bytes 255 heads, 63 sectors/track, 354942 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdr doesn't contain a valid partition table
Disk /dev/sds: 322.1 GB, 322122547200 bytes 255 heads, 63 sectors/track, 39162 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sds1 1 39162 314568733+ 8e Linux LVM
Disk /dev/sdt: 322.1 GB, 322122547200 bytes 255 heads, 63 sectors/track, 39162 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System /dev/sdt1 1 39162 314568733+ 8e Linux LVM
5 創建新的OCRNEW磁盤組 su – grid sqlplus / as sysasm CREATE DISKGROUP OCRNEW NORMAL REDUNDANCY DISK 'ORCL:OCR_VOL4' NAME VOL4 DISK 'ORCL:OCR_VOL5' NAME VOL5 DISK 'ORCL:OCR_VOL6' NAME VOL6 ATTRIBUTE 'compatible.asm'='11.2';
6 添加OCR信息到OCRNEW [root@oracle1 bin]# ./ocrconfig -add +OCRNEW [root@oracle1 bin]# ./ocrcheck -config Oracle Cluster Registry configuration is : Device/File Name : +OCR Device/File Name : +OCRNEW [root@oracle1 bin]# more /etc/oracle/ocr.loc #Device/file getting replaced by device +OCRNEW ocrconfig_loc=+OCR ocrmirrorconfig_loc=+OCRNEW local_only=false [root@oracle1 bin]#
可以看到OCRNEW 磁盤組已經成功添加到OCR磁盤信息中
遷移vote 文件 當前votedisk信息 [grid@oracle1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 14f694d9d4414f9ebf85d3ce6b9aef0b (ORCL:OCR_VOL1) [OCR] 2. ONLINE 9f9ee7281c954f8abfcc6e88c33257ac (ORCL:OCR_VOL2) [OCR] 3. ONLINE 38114fd602194fa9bf4d05655b3d89b7 (ORCL:OCR_VOL3) [OCR] Located 3 voting disk(s).
[grid@oracle1 ~]$ crsctl replace votedisk +OCRNEW Successful addition of voting disk 00634ef593ee4f92bf48e8c089cb5565. Successful addition of voting disk 232159722de04f67bf03a78b757e3bec. Successful addition of voting disk a340d5b23aac4f6fbf9f7b1d59088fa5. Successful deletion of voting disk 14f694d9d4414f9ebf85d3ce6b9aef0b. Successful deletion of voting disk 9f9ee7281c954f8abfcc6e88c33257ac. Successful deletion of voting disk 38114fd602194fa9bf4d05655b3d89b7. Successfully replaced voting disk group with +OCRNEW. CRS-4266: Voting file(s) successfully replaced
SQL> create spfile='+OCRNEW' from pfile='/home/grid/asmpfile.ora'; File created.
8 刪除ASM磁盤組OCR [root@oracle1 bin]# ./ocrconfig -delete +OCR 查看OCR和VOTE新狀態與位置 [root@oracle1 bin]# ./ocrcheck && ./crsctl query css votedisk Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2768 Available space (kbytes) : 259352 ID : 2006438789 Device/File Name : +OCRNEW Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded
## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 00634ef593ee4f92bf48e8c089cb5565 (ORCL:OCR_VOL4) [OCRNEW] 2. ONLINE 232159722de04f67bf03a78b757e3bec (ORCL:OCR_VOL5) [OCRNEW] 3. ONLINE a340d5b23aac4f6fbf9f7b1d59088fa5 (ORCL:OCR_VOL6) [OCRNEW] Located 3 voting disk(s).
SYS@+ASM1> alter diskgroup OCR dismount; Diskgroup altered. SYS@+ASM2> drop diskgroup OCR including contents; Diskgroup dropped. SYS@+ASM2> SQL> select GROUP_NUMBER,NAME,STATE,type,TOTAL_MB,free_mb,VOTING_FILES,COMPATIBILITY from v$asm_diskgroup; GROUP_NUMBER NAME STATE TYPE TOTAL_MB FREE_MB V COMPATIBILITY ------------ ------------------------------ ----------- ------ ---------- ---------- - ------------------------------------------------------------ 1 BACK MOUNTED NORMAL 3071982 3070675 N 11.2.0.0.0 2 DATA MOUNTED NORMAL 4095976 1561759 N 11.2.0.0.0 3 OCRNEW MOUNTED NORMAL 102396 101470 N 11.2.0.0.0
SYS@+ASM2> SQL> select GROUP_NUMBER,DISK_NUMBER,STATE,REDUNDANCY,TOTAL_MB,FREE_MB,name,path,failgroup from v$asm_disk order by GROUP_NUMBER; GROUP_NUMBER DISK_NUMBER STATE REDUNDA TOTAL_MB FREE_MB NAME PATH FAILGROUP ------------ ----------- -------- ------- ---------- ---------- ------------------------------ ------------------------------ ------------------------------ 0 0 NORMAL UNKNOWN 0 0 ORCL:OCR_VOL1 0 1 NORMAL UNKNOWN 0 0 ORCL:OCR_VOL2 0 2 NORMAL UNKNOWN 0 0 ORCL:OCR_VOL3 1 1 NORMAL UNKNOWN 1023994 1023559 DATA_VOL5 ORCL:DATA_VOL5 DATA_VOL5 1 0 NORMAL UNKNOWN 1023994 1023559 DATA_VOL4 ORCL:DATA_VOL4 DATA_VOL4 1 2 NORMAL UNKNOWN 1023994 1023557 DATA_VOL6 ORCL:DATA_VOL6 DATA_VOL6 2 2 NORMAL UNKNOWN 1023994 390447 DATA_VOL2 ORCL:DATA_VOL2 DATA_VOL2 2 1 NORMAL UNKNOWN 1023994 390450 DATA_VOL1 ORCL:DATA_VOL1 DATA_VOL1 2 0 NORMAL UNKNOWN 1023994 390436 BACK_VOL1 ORCL:BACK_VOL1 BACK_VOL1 2 3 NORMAL UNKNOWN 1023994 390426 DATA_VOL3 ORCL:DATA_VOL3 DATA_VOL3 3 0 NORMAL UNKNOWN 31376 31075 VOL4 ORCL:OCR_VOL4 VOL4
GROUP_NUMBER DISK_NUMBER STATE REDUNDA TOTAL_MB FREE_MB NAME PATH FAILGROUP ------------ ----------- -------- ------- ---------- ---------- ------------------------------ ------------------------------ ------------------------------ 3 1 NORMAL UNKNOWN 31376 31077 VOL5 ORCL:OCR_VOL5 VOL5 3 2 NORMAL UNKNOWN 39644 39318 VOL6 ORCL:OCR_VOL6 VOL6 13 rows selected. 至此整個OCR&VOTING遷移過程結束
9.這里可以重啟下集群crs測試OCR&VOTE是否遷移成功,當然你也可以選擇不重啟,個人建議重啟測試下。 查看OCR&VOTE位置及ASM實例spfile位置 [root@oracle1 bin]# ./ocrcheck && ./crsctl query css votedisk Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 2768 Available space (kbytes) : 259352 ID : 2006438789 Device/File Name : +OCRNEW Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded
## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 00634ef593ee4f92bf48e8c089cb5565 (ORCL:OCR_VOL4) [OCRNEW] 2. ONLINE 232159722de04f67bf03a78b757e3bec (ORCL:OCR_VOL5) [OCRNEW] 3. ONLINE a340d5b23aac4f6fbf9f7b1d59088fa5 (ORCL:OCR_VOL6) [OCRNEW] Located 3 voting disk(s).
SQL> show parameter spfile; NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ spfile string +OCRNEW/oracle-cluster/asmpara meterfile/registry.253.8456918 87 SQL> 10 遷移數據磁盤組數據 SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL01' rebalance power 11; Diskgroup altered. SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL02' rebalance power 11; Diskgroup altered. SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL03' rebalance power 11; Diskgroup altered. SQL> alter diskgroup DATA add disk 'ORCL:DATA_VOL04' rebalance power 11; Diskgroup altered. SQL> alter diskgroup back add disk 'ORCL:DATA_VOL05' rebalance power 11; Diskgroup altered. SQL> alter diskgroup back add disk 'ORCL:DATA_VOL06' rebalance power 11; SQL> alter diskgroup BACK add disk 'ORCL:BACK_VOL01' rebalance power 11; Diskgroup altered. 由于指定rebalance power 11,ASM會自動均衡ASM磁盤組DATA里面存放的數據在各ASM磁盤的分布。
當rebalance結束后,查詢V$ASM_OPERATION視圖,將不會返回信息。 SQL> select * from V$ASM_OPERATION; no rows selected
11 刪除數據磁盤組上的舊磁盤 alter diskgroup data drop disk 'BACK_VOL1' rebalance power 11; alter diskgroup data drop disk 'DATA_VOL2' rebalance power 11; alter diskgroup data drop disk 'DATA_VOL3' rebalance power 11; alter diskgroup back drop disk 'DATA_VOL4' rebalance power 11; alter diskgroup back drop disk 'DATA_VOL5' rebalance power 11; alter diskgroup back drop disk 'DATA_VOL6' rebalance power 11; ASM不但向磁盤組中加入新磁盤時會做rebalance,在刪除ASM磁盤時也會rebalance,將該磁盤上的數據rebalance到其他該磁盤組的磁盤中。
按照此方法刪除ASM磁盤后,ASM的所有數據都已經存放在新的存儲上。 09:40:38 SQL> select a.NAME GROUP_NAME,a.TOTAL_MB,a.FREE_MB GROUP_FREE_MB,b.OS_MB,b.FREE_MB,b.name,b.path from v$asm_diskgroup a,v$asm_disk b where a.GROUP_NUMBER=b.GROUP_NUMBER; GROUP_NAME TOTAL_MB GROUP_FREE_MB OS_MB FREE_MB NAME PATH ------------------------------ ---------- ------------- ---------- ---------- ------------------------------ ---------------------------------------- BACK 3071982 3070868 1023994 1023622 DATA_VOL05 ORCL:DATA_VOL05 BACK 3071982 3070868 1023994 1023624 DATA_VOL06 ORCL:DATA_VOL06 OCRNEW 102396 101470 31376 31075 VOL4 ORCL:OCR_VOL4 OCRNEW 102396 101470 31376 31077 VOL5 ORCL:OCR_VOL5 OCRNEW 102396 101470 39644 39318 VOL6 ORCL:OCR_VOL6 DATA 4095976 1561759 1023994 390437 DATA_VOL01 ORCL:DATA_VOL01 DATA 4095976 1561759 1023994 390440 DATA_VOL02 ORCL:DATA_VOL02 DATA 4095976 1561759 1023994 390443 DATA_VOL03 ORCL:DATA_VOL03 DATA 4095976 1561759 1023994 390439 DATA_VOL04 ORCL:DATA_VOL04 BACK 3071982 3070868 1023994 1023622 BACK_VOL01 ORCL:BACK_VOL01
[root@oracle2 bin]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Cleaning disk "BACK_VOL1" Cleaning disk "DATA_VOL1" Cleaning disk "DATA_VOL2" Cleaning disk "DATA_VOL3" Cleaning disk "DATA_VOL4" Cleaning disk "DATA_VOL5" Cleaning disk "DATA_VOL6" Scanning system for ASM disks... You have new mail in /var/spool/mail/root