91超碰碰碰碰久久久久久综合_超碰av人澡人澡人澡人澡人掠_国产黄大片在线观看画质优化_txt小说免费全本

溫馨提示×

溫馨提示×

您好,登錄后才能下訂單哦!

密碼登錄×
登錄注冊×
其他方式登錄
點擊 登錄注冊 即表示同意《億速云用戶服務條款》

Linux udev 動態管理Oracle ASM磁盤-實驗

發布時間:2020-08-10 20:37:03 來源:ITPUB博客 閱讀:212 作者:SmartWilson 欄目:建站服務器

1       概述

Red Hat Enterprise Linux (RHEL) 6以前,Oracle都是采用ASMLib包來管理ASM磁盤。但是20115月甲骨文發表了一份聲明,聲明中稱Oracle將不再提供Red Hat Enterprise Linux (RHEL) 6ASMLib包和相關更新。

甲骨文在這份聲明中表示,ASMLib更新將通過Unbreakable Linux NetworkULN)來發布,并僅對Oracle Linux客戶開放。ULN雖然為甲骨文和紅帽的客戶服務,但如果客戶想要使用ASMLib,就必須使用Oraclekernel來替換掉紅帽的kernel

 

本實驗部署Oracle+ASM,不再采用ASMLib,而是采用udev動態管理ASM磁盤。

 

注意:Linux 內核 2.6以上才支持udev

 

虛擬機:VMware Workstation 9

操作系統:Red Hat Enterprise Linux Server release 5.6

系統內核:2.6.18-238.el5

數據庫:Oracle Database 11.2.0.1

 

2       實驗步驟

之前已經基于udev的方式創建過ASM磁盤,而且已經安裝好Oracle Grid Oracle Database,并創建好了數據庫,接下來將添加41G磁盤,分別擴展現有ASM磁盤組DG_DATA,新建ASM磁盤組DG_NEW_DATA

 

查看新添加的磁盤,分別為/dev/sdb/dev/sdg/dev/sdh/dev/sdi

 

[root@ora32 dev]# fdisk -l

 

Disk /dev/sda: 23.6 GB, 23622320128 bytes

255 heads, 63 sectors/track, 2871 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *           1          65      522081   83  Linux

/dev/sda2              66         457     3148740   82  Linux swap / Solaris

/dev/sda3             458        2871    19390455   8e  Linux LVM

 

Disk /dev/sdb: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

Disk /dev/sdb doesn't contain a valid partition table

 

Disk /dev/sdc: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdc1               1         261     2096451   83  Linux

 

Disk /dev/sdd: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdd1               1         261     2096451   83  Linux

 

Disk /dev/sde: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sde1               1         261     2096451   83  Linux

 

Disk /dev/sdf: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdf1               1         261     2096451   83  Linux

 

Disk /dev/sdg: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

Disk /dev/sdg doesn't contain a valid partition table

 

Disk /dev/sdh: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

Disk /dev/sdh doesn't contain a valid partition table

 

Disk /dev/sdi: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

Disk /dev/sdi doesn't contain a valid partition table

 

 

 

分別對4塊磁盤進行分區,每塊磁盤分1個區即可。

[root@ora32 dev]# fdisk -l

 

Disk /dev/sda: 23.6 GB, 23622320128 bytes

255 heads, 63 sectors/track, 2871 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sda1   *           1          65      522081   83  Linux

/dev/sda2              66         457     3148740   82  Linux swap / Solaris

/dev/sda3             458        2871    19390455   8e  Linux LVM

 

Disk /dev/sdb: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdb1               1         130     1044193+  83  Linux

 

Disk /dev/sdc: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdc1               1         261     2096451   83  Linux

 

Disk /dev/sdd: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdd1               1         261     2096451   83  Linux

 

Disk /dev/sde: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sde1               1         261     2096451   83  Linux

 

Disk /dev/sdf: 2147 MB, 2147483648 bytes

255 heads, 63 sectors/track, 261 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdf1               1         261     2096451   83  Linux

 

Disk /dev/sdg: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdg1               1         130     1044193+  83  Linux

 

Disk /dev/sdh: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdh2               1         130     1044193+  83  Linux

 

Disk /dev/sdi: 1073 MB, 1073741824 bytes

255 heads, 63 sectors/track, 130 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

 

   Device Boot      Start         End      Blocks   Id  System

/dev/sdi1               1         130     1044193+  83  Linux

 

現在需要通過udev來管理這幾個磁盤設備文件,通過規則文件將磁盤映射為固定名稱,并賦予所屬用戶,用戶組,操作權限等。

查看udev規則文件

[root@ora32 dev]# cd /etc/udev/rules.d

[root@ora32 rules.d]# ll

total 248

-rw-r--r-- 1 root root   515 Nov 25  2010 05-udev-early.rules

-rw-r--r-- 1 root root   948 Dec  8  2010 40-multipath.rules

-rw-r--r-- 1 root root 16153 Nov 25  2010 50-udev.rules

-rw-r--r-- 1 root root   163 Dec 11  2010 51-dlm.rules

-rw-r--r-- 1 root root   471 Nov 25  2010 51-hotplug.rules

-rw-r--r-- 1 root root   138 May 10 17:28 55-usm.rules

-rw-r--r-- 1 root root 58016 Oct 13  2006 60-libsane.rules

-rw-r--r-- 1 root root   143 Oct 13  2010 60-net.rules

-rw-r--r-- 1 root root  1088 Jul 18  2006 60-pcmcia.rules

-rw-r--r-- 1 root root   316 Sep 22  2010 60-raw.rules

-rw-r--r-- 1 root root  9175 Nov 13  2010 60-wacom.rules

-rw-r--r-- 1 root root   129 Nov 30  2010 61-uinput-stddev.rules

-rw-r--r-- 1 root root   214 Nov 30  2010 61-uinput-wacom.rules

-rw-r--r-- 1 root root  1823 Sep 19  2008 85-pcscd_ccid.rules

-rw-r--r-- 1 root root   114 Aug 29  2008 90-alsa.rules

-rw-r--r-- 1 root root    61 Nov 25  2010 90-dm.rules

-rw-r--r-- 1 root root    82 Nov  4  2010 90-hal.rules

-rw-r--r-- 1 root root   107 Nov 25  2010 95-pam-console.rules

-rw-r--r-- 1 root root   292 Nov 30  2010 98-kexec.rules

-rw-r--r-- 1 root root   669 May 13 13:51 99-oracle-asmdevices.rules

-rw-r--r-- 1 root root    60 Sep 16  2012 99-oracle.rules

-rw-r--r-- 1 root root   334 Sep 15  2012 99-vmware-scsi-udev.rules

-rw-r--r-- 1 root root  2319 Jul  9  2008 bluetooth.rules

 

udev規則文件“99-oracle-asmdevices.rules”之前已經創建好,看一下當前的映射規則和ASM磁盤情況。

[root@ora32 rules.d]# cat 99-oracle-asmdevices.rules

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29c9a269653c7cd04ea2edb0c71", NAME="ASM_DISK_2G_1", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29861258ebc95eded634a1f36fd", NAME="ASM_DISK_2G_2", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c296fd09d9ed213df1999820ca89", NAME="ASM_DISK_2G_3", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c2934ae12fd4af0dd366d85b42f5",  NAME="ASM_DISK_2G_4", WNER="grid", GROUP="asmadmin", MODE="0660"

 

規則解釋,以第一條為例。

系統啟動時,udev根據系統總線scsiBUS=="scsi")和內核設備文件名sd?1KERNEL=="sd?1"),過濾出需要動態映射的設備,然后再利用/sbin/scsi_idPROGRAM=="/sbin/scsi_id")命令輸出的SCSI磁盤id,輸出結果中匹配上"36000c29c9a269653c7cd04ea2edb0c71"SCSI磁盤,則會應用后續的命名、授權等操作(NAME="ASM_DISK_2G_1", WNER="grid", GROUP="asmadmin", MODE="0660"),SCSI磁盤ID可以通過/sbin/scsi_id命令獲取。

 

[root@ora32 rules.d]# /sbin/scsi_id -g -s /block/sdc/sdc1

36000c29c9a269653c7cd04ea2edb0c71

[root@ora32 rules.d]# /sbin/scsi_id -g -s /block/sdd/sdd1

36000c29861258ebc95eded634a1f36fd

[root@ora32 rules.d]# /sbin/scsi_id -g -s /block/sde/sde1

36000c296fd09d9ed213df1999820ca89

[root@ora32 rules.d]# /sbin/scsi_id -g -s /block/sdf/sdf1

36000c2934ae12fd4af0dd366d85b42f5

 

從上述結果可以看出,udev已經做了如下映射關系。

/block/sdc/sdc1 => ASM_DISK_2G_1

/block/sdd/sdd1 => ASM_DISK_2G_2

/block/sde/sde1 => ASM_DISK_2G_3

/block/sdf/sdf1 => ASM_DISK_2G_4

 

[grid@ora32 dev]$ ll ASM*

brw-rw---- 1 grid asmadmin 8, 33 May 13 14:57 ASM_DISK_2G_1

brw-rw---- 1 grid asmadmin 8, 49 May 13 14:57 ASM_DISK_2G_2

brw-rw---- 1 grid asmadmin 8, 65 May 13 14:54 ASM_DISK_2G_3

brw-rw---- 1 grid asmadmin 8, 81 May 13 14:57 ASM_DISK_2G_4

 

通過Oracle視圖看一下當前磁盤組、磁盤的使用情況。

[grid@ora32 ~]$ sqlplus / as sysasm

 

SQL*Plus: Release 11.2.0.1.0 Production on Mon May 13 14:54:01 2013

 

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

 

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Automatic Storage Management option

 

SQL>

SQL>

SQL> col name for a10

SQL> SELECT group_number,name,type,state,total_mb,free_mb,offline_disks FROM V$ASM_diskgroup;

 

GROUP_NUMBER NAME       TYPE   STATE         TOTAL_MB    FREE_MB OFFLINE_DISKS

------------ ---------- ------ ----------- ---------- ---------- -------------

           1 DG_DATA    EXTERN MOUNTED           8188       6448             0

 

 

SQL> col group# for 99

SQL> col disk# for 99

SQL> col name for a5

SQL> col path for a11

SQL> col LABEL for a7

SQL> col FAILGROUP for a10

SQL> select GROUP_NUMBER group#,DISK_NUMBER disk#,LABEL,STATE,FAILGROUP,name,path,TOTAL_MB,FREE_MB from v$asm_disk;

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

     1     2         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1612

                              02         TA_00 SK_2G_3

                                         02

 

     1     3         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1611

                              03         TA_00 SK_2G_4

                                         03

 

     1     1         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1613

                              01         TA_00 SK_2G_2

                                         01

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

 

     1     0         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1612

                              00         TA_00 SK_2G_1

                                         00

 

現將新增的4塊硬盤通過udev映射為需要的設備文件,然后將其中2塊增加到ASM磁盤組DG_DATA中,另外2塊則新建一個ASM磁盤組DG_NEW_DATA

 

[root@ora32 rules.d]# /sbin/scsi_id -g -s /block/sdb/sdb1

36000c2968513d97eb7cdd25bdbd10600

[root@ora32 rules.d]# /sbin/scsi_id -g -s /block/sdg/sdg1

36000c29682ba5ec7a4c8a189499a89bb

[root@ora32 rules.d]# /sbin/scsi_id -g -s /block/sdh/sdh2

36000c292aa4a4ebbb25d15d2084c01ca

[root@ora32 rules.d]# /sbin/scsi_id -g -s /blokc/sdi/sdi1

 

新增設備映射規則

[root@ora32 rules.d]# vi 99-oracle-asmdevices.rules

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29c9a269653c7cd04ea2edb0c71", NAME="ASM_DISK_2G_1", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29861258ebc95eded634a1f36fd", NAME="ASM_DISK_2G_2", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c296fd09d9ed213df1999820ca89", NAME="ASM_DISK_2G_3", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c2934ae12fd4af0dd366d85b42f5",  NAME="ASM_DISK_2G_4", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c2968513d97eb7cdd25bdbd10600",  NAME="ASM_DISK_1G_1", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29682ba5ec7a4c8a189499a89bb",  NAME="ASM_DISK_1G_2", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c292aa4a4ebbb25d15d2084c01ca",  NAME="ASM_DISK_1G_3", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29a98ca1bd8a0012e99f992a19d",  NAME="ASM_DISK_1G_4", WNER="grid", GROUP="asmadmin", MODE="0660"

 

 

如果是第一次配置udev映射關系,需要如下操作。

1、  在目錄/etc/udev/rules.d/中新建99-oracle-asmdevices.rules文件。

2、配置SCSI 設備為可信任設備,配置/etc/scsi_id.config文件,添加options=-g配置項。

[root@ora32 rules.d]# cat /etc/scsi_id.config

#

# scsi_id configuration

#

# lower or upper case has no affect on the left side. Quotes (") are

# required for spaces in values. Model is the same as the SCSI

# INQUIRY product identification field. Per the SCSI INQUIRY, the vendor

# is limited to 8 bytes, model to 16 bytes.

#

# The first matching line found is used. Short matches match longer ones,

# if you do not want such a match space fill the extra bytes. If no model

# is specified, only the vendor string need match.

#

# ptions=<any scsi_id command line options>

# vendor=string[,model=string],options=<per-device scsi_id options>

 

# some libata drives require vpd page 0x80

vendor="ATA",options=-p 0x80

options=-g

 

測試udev能否正確映射

[root@ora32 ~]# testudev /block/sdg/sdg1

-bash: testudev: command not found

[root@ora32 ~]# udevtest /block/sdb/sdb1

main: looking at device '/block/sdb/sdb1' from subsystem 'block'

udev_rules_get_name: add symlink 'disk/by-id/scsi-36000c2968513d97eb7cdd25bdbd10600-part1'

udev_rules_get_name: add symlink 'disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0-part1'

run_program: '/lib/udev/vol_id --export /dev/.tmp-8-17'

run_program: '/lib/udev/vol_id' returned with status 4

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

udev_rules_get_name: rule applied, 'sdb1' becomes 'ASM_DISK_1G_1'

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c2968513d97eb7cdd25bdbd10600'

run_program: '/sbin/scsi_id' returned with status 0

udev_device_event: device '/block/sdb/sdb1' already in database, validate currently present symlinks

udev_node_add: creating device node '/dev/ASM_DISK_1G_1', major = '8', minor = '17', mode = '0660', uid = '503', gid = '504'

udev_node_add: creating symlink '/dev/disk/by-id/scsi-36000c2968513d97eb7cdd25bdbd10600-part1' to '../../ASM_DISK_1G_1'

udev_node_add: creating symlink '/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0-part1' to '../../ASM_DISK_1G_1'

main: run: 'socket:/org/kernel/udev/monitor'

main: run: '/lib/udev/udev_run_devd'

main: run: 'socket:/org/freedesktop/hal/udev_event'

main: run: '/sbin/pam_console_apply /dev/ASM_DISK_1G_1 /dev/disk/by-id/scsi-36000c2968513d97eb7cdd25bdbd10600-part1 /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:1:0-part1'

main: run: '/bin/sh -c 'echo 180 >/sys/block/sdb/sdb1/timeout''

 

 

 

[root@ora32 ~]# udevtest /block/sdg/sdg1

main: looking at device '/block/sdg/sdg1' from subsystem 'block'

udev_rules_get_name: add symlink 'disk/by-id/scsi-36000c29682ba5ec7a4c8a189499a89bb-part1'

udev_rules_get_name: add symlink 'disk/by-path/pci-0000:00:10.0-scsi-0:0:6:0-part1'

run_program: '/lib/udev/vol_id --export /dev/.tmp-8-97'

run_program: '/lib/udev/vol_id' returned with status 4

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

udev_rules_get_name: rule applied, 'sdg1' becomes 'ASM_DISK_1G_2'

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29682ba5ec7a4c8a189499a89bb'

run_program: '/sbin/scsi_id' returned with status 0

udev_device_event: device '/block/sdg/sdg1' already in database, validate currently present symlinks

udev_node_add: creating device node '/dev/ASM_DISK_1G_2', major = '8', minor = '97', mode = '0660', uid = '503', gid = '504'

udev_node_add: creating symlink '/dev/disk/by-id/scsi-36000c29682ba5ec7a4c8a189499a89bb-part1' to '../../ASM_DISK_1G_2'

udev_node_add: creating symlink '/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:6:0-part1' to '../../ASM_DISK_1G_2'

main: run: 'socket:/org/kernel/udev/monitor'

main: run: '/lib/udev/udev_run_devd'

main: run: 'socket:/org/freedesktop/hal/udev_event'

main: run: '/sbin/pam_console_apply /dev/ASM_DISK_1G_2 /dev/disk/by-id/scsi-36000c29682ba5ec7a4c8a189499a89bb-part1 /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:6:0-part1'

main: run: '/bin/sh -c 'echo 180 >/sys/block/sdg/sdg1/timeout''

 

 

 

[root@ora32 ~]# udevtest /block/sdh/sdh2

main: looking at device '/block/sdh/sdh2' from subsystem 'block'

udev_rules_get_name: add symlink 'disk/by-id/scsi-36000c292aa4a4ebbb25d15d2084c01ca-part1'

udev_rules_get_name: add symlink 'disk/by-path/pci-0000:00:10.0-scsi-0:0:8:0-part1'

run_program: '/lib/udev/vol_id --export /dev/.tmp-8-113'

run_program: '/lib/udev/vol_id' returned with status 4

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

udev_rules_get_name: rule applied, 'sdh2' becomes 'ASM_DISK_1G_3'

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c292aa4a4ebbb25d15d2084c01ca'

run_program: '/sbin/scsi_id' returned with status 0

udev_device_event: device '/block/sdh/sdh2' already in database, validate currently present symlinks

udev_node_add: creating device node '/dev/ASM_DISK_1G_3', major = '8', minor = '113', mode = '0660', uid = '503', gid = '504'

udev_node_add: creating symlink '/dev/disk/by-id/scsi-36000c292aa4a4ebbb25d15d2084c01ca-part1' to '../../ASM_DISK_1G_3'

udev_node_add: creating symlink '/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:8:0-part1' to '../../ASM_DISK_1G_3'

main: run: 'socket:/org/kernel/udev/monitor'

main: run: '/lib/udev/udev_run_devd'

main: run: 'socket:/org/freedesktop/hal/udev_event'

main: run: '/sbin/pam_console_apply /dev/ASM_DISK_1G_3 /dev/disk/by-id/scsi-36000c292aa4a4ebbb25d15d2084c01ca-part1 /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:8:0-part1'

main: run: '/bin/sh -c 'echo 180 >/sys/block/sdh/sdh2/timeout''

[root@ora32 ~]# udevtest /block/sdi/sdi1

main: looking at device '/block/sdi/sdi1' from subsystem 'block'

udev_rules_get_name: add symlink 'disk/by-id/scsi-36000c29a98ca1bd8a0012e99f992a19d-part1'

udev_rules_get_name: add symlink 'disk/by-path/pci-0000:00:10.0-scsi-0:0:9:0-part1'

run_program: '/lib/udev/vol_id --export /dev/.tmp-8-129'

run_program: '/lib/udev/vol_id' returned with status 4

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

run_program: '/sbin/scsi_id'

run_program: '/sbin/scsi_id' (stdout) '36000c29a98ca1bd8a0012e99f992a19d'

run_program: '/sbin/scsi_id' returned with status 0

udev_rules_get_name: rule applied, 'sdi1' becomes 'ASM_DISK_1G_4'

udev_device_event: device '/block/sdi/sdi1' already in database, validate currently present symlinks

udev_node_add: creating device node '/dev/ASM_DISK_1G_4', major = '8', minor = '129', mode = '0660', uid = '503', gid = '504'

udev_node_add: creating symlink '/dev/disk/by-id/scsi-36000c29a98ca1bd8a0012e99f992a19d-part1' to '../../ASM_DISK_1G_4'

udev_node_add: creating symlink '/dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:9:0-part1' to '../../ASM_DISK_1G_4'

main: run: 'socket:/org/kernel/udev/monitor'

main: run: '/lib/udev/udev_run_devd'

main: run: 'socket:/org/freedesktop/hal/udev_event'

main: run: '/sbin/pam_console_apply /dev/ASM_DISK_1G_4 /dev/disk/by-id/scsi-36000c29a98ca1bd8a0012e99f992a19d-part1 /dev/disk/by-path/pci-0000:00:10.0-scsi-0:0:9:0-part1'

main: run: '/bin/sh -c 'echo 180 >/sys/block/sdi/sdi1/timeout''

 

 

重新啟動udev服務

[root@ora32 ~]# /sbin/udevcontrol reload_rules

[root@ora32 ~]# /sbin/start_udev

Starting udev: [  OK  ]

 

查看一下新映射的磁盤

[root@ora32 ~]# ll /dev/ASM*

brw-rw---- 1 grid asmadmin 8,  17 May 13 15:40 /dev/ASM_DISK_1G_1

brw-rw---- 1 grid asmadmin 8,  97 May 13 15:40 /dev/ASM_DISK_1G_2

brw-rw---- 1 grid asmadmin 8, 113 May 13 15:40 /dev/ASM_DISK_1G_3

brw-rw---- 1 grid asmadmin 8, 129 May 13 15:40 /dev/ASM_DISK_1G_4

brw-rw---- 1 grid asmadmin 8,  33 May 13 15:43 /dev/ASM_DISK_2G_1

brw-rw---- 1 grid asmadmin 8,  49 May 13 15:43 /dev/ASM_DISK_2G_2

brw-rw---- 1 grid asmadmin 8,  65 May 13 15:42 /dev/ASM_DISK_2G_3

brw-rw---- 1 grid asmadmin 8,  81 May 13 15:43 /dev/ASM_DISK_2G_4

 

通過Oracle視圖查看一下ASM磁盤情況

[grid@ora32 ~]$ sqlplus / as sysasm

 

SQL*Plus: Release 11.2.0.1.0 Production on Mon May 13 15:42:05 2013

 

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

 

 

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production

With the Automatic Storage Management option

 

SQL>

SQL> col group# for 99

SQL> col disk# for 99

SQL> col name for a5

SQL> col path for a11

SQL> col LABEL for a7

SQL> col FAILGROUP for a10

SQL> select GROUP_NUMBER group#,DISK_NUMBER disk#,LABEL,STATE,FAILGROUP,name,path,TOTAL_MB,FREE_MB from v$asm_disk;

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

     0     0         NORMAL                    /dev/ASM_DI          0          0

                                               SK_1G_3

 

     0     1         NORMAL                    /dev/ASM_DI          0          0

                                               SK_1G_2

 

     0     2         NORMAL                    /dev/ASM_DI          0          0

                                               SK_1G_1

 

     0     3         NORMAL                    /dev/ASM_DI          0          0

                                               SK_1G_4

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

 

     1     2         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1612

                              02         TA_00 SK_2G_3

                                         02

 

     1     3         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1611

                              03         TA_00 SK_2G_4

                                         03

 

     1     1         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1613

                              01         TA_00 SK_2G_2

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

                                         01

 

     1     0         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1612

                              00         TA_00 SK_2G_1

                                         00

 

 

8 rows selected.

此時發現新添加的4塊硬盤,已經可以通過視圖進行查詢,但是FAILGROUPNAME等屬性為空,因為磁盤還沒有加到ASM磁盤組。ASM是怎么認到新增加的磁盤?其實它是根據參數文件中的屬性asm_diskstring自動掃描到的,這個參數是在安裝GRID同時配置ASM時設置的。

SQL> show parameter ASM

 

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

asm_diskgroups                       string

asm_diskstring                       string      /dev/ASM*

asm_power_limit                      integer     1

asm_preferred_read_failure_groups    string

 

 

 

將磁盤ASM_DISK_1G_1ASM_DISK_1G_2添加到磁盤組DG_DATA中。

SQL> ALTER DISKGROUP DG_DATA ADD disk '/dev/ASM_DISK_1G_1';

 

Diskgroup altered.

 

SQL> ALTER DISKGROUP DG_DATA ADD disk '/dev/ASM_DISK_1G_2';

 

Diskgroup altered.

 

新建磁盤組DG_NEW_DATA,將ASM_DISK_1G_3ASM_DISK_1G_4添加到磁盤組中。

SQL> CREATE DISKGROUP DG_NEW_DATA external REDUNDANCY DISK '/dev/ASM_DISK_1G_3','/dev/ASM_DISK_1G_4';

 

Diskgroup created.

 

通過Oracle 視圖查看一下新增ASM磁盤組、ASM磁盤的情況

SQL> col name for a10

SQL> SELECT group_number,name,type,state,total_mb,free_mb,offline_disks FROM V$ASM_diskgroup;

 

GROUP_NUMBER NAME       TYPE   STATE         TOTAL_MB    FREE_MB OFFLINE_DISKS

------------ ---------- ------ ----------- ---------- ---------- -------------

           1 DG_DATA    EXTERN MOUNTED          10226       8482             0

           2 DG_NEW_DAT EXTERN MOUNTED           2038       1986             0

             A

 

SQL> select GROUP_NUMBER group#,DISK_NUMBER disk#,LABEL,STATE,FAILGROUP,name,path,TOTAL_MB,FREE_MB from v$asm_disk;

 

    GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB

---------- ----- ------- -------- ---------- ----- ----------- ----------

   FREE_MB

----------

         1     2         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047

                                  02         TA_00 SK_2G_3

                                             02

      1697

 

         1     3         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047

                                  03         TA_00 SK_2G_4

                                             03

      1697

 

    GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB

---------- ----- ------- -------- ---------- ----- ----------- ----------

   FREE_MB

----------

 

         1     1         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047

                                  01         TA_00 SK_2G_2

                                             01

      1699

 

         1     0         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047

                                  00         TA_00 SK_2G_1

                                             00

 

    GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB

---------- ----- ------- -------- ---------- ----- ----------- ----------

   FREE_MB

----------

      1697

 

         2     0         NORMAL   DG_NEW_DAT DG_NE /dev/ASM_DI       1019

                                  A_0000     W_DAT SK_1G_3

                                             A_000

                                             0

       994

 

         1     5         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       1019

 

    GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB

---------- ----- ------- -------- ---------- ----- ----------- ----------

   FREE_MB

----------

                                  05         TA_00 SK_1G_2

                                             05

       846

 

         1     4         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       1019

                                  04         TA_00 SK_1G_1

                                             04

       846

 

 

    GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB

---------- ----- ------- -------- ---------- ----- ----------- ----------

   FREE_MB

----------

         2     1         NORMAL   DG_NEW_DAT DG_NE /dev/ASM_DI       1019

                                  A_0001     W_DAT SK_1G_4

                                             A_000

                                             1

       992

 

 

8 rows selected.

 

 

根據實驗要求4塊硬盤添加完畢,接下來將硬盤ASM_DISK_1G_1ASM_DISK_1G_2ASM磁盤組DG_DATA中刪除,然后將udev映射規則刪除。

SQL> ALTER DISKGROUP DG_DATA DROP DISK DG_DATA_0004;

 

Diskgroup altered.

 

SQL> ALTER DISKGROUP DG_DATA DROP DISK DG_DATA_0005;

 

Diskgroup altered.

 

 

 

 

SQL> col group# for 99

SQL> col disk# for 99

SQL> col name for a5

SQL> col path for a11

SQL> col LABEL for a7

SQL> col FAILGROUP for a10

SQL> select GROUP_NUMBER group#,DISK_NUMBER disk#,LABEL,STATE,FAILGROUP,name,path,TOTAL_MB,FREE_MB from v$asm_disk;

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

     0     0         NORMAL                    /dev/ASM_DI          0          0

                                               SK_1G_2

 

     0     1         NORMAL                    /dev/ASM_DI          0          0

                                               SK_1G_1

 

     1     2         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1612

                              02         TA_00 SK_2G_3

                                         02

 

     1     3         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1611

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

                              03         TA_00 SK_2G_4

                                         03

 

     1     1         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1614

                              01         TA_00 SK_2G_2

                                         01

 

     1     0         NORMAL   DG_DATA_00 DG_DA /dev/ASM_DI       2047       1611

                              00         TA_00 SK_2G_1

                                         00

 

 

GROUP# DISK# LABEL   STATE    FAILGROUP  NAME  PATH          TOTAL_MB    FREE_MB

------ ----- ------- -------- ---------- ----- ----------- ---------- ----------

     2     0         NORMAL   DG_NEW_DAT DG_NE /dev/ASM_DI       1019        994

                              A_0000     W_DAT SK_1G_3

                                         A_000

                                         0

 

     2     1         NORMAL   DG_NEW_DAT DG_NE /dev/ASM_DI       1019        992

                              A_0001     W_DAT SK_1G_4

                                         A_000

                                         1

 

 

8 rows selected.

 

 

 

[root@ora32 ~]# vi /etc/udev/rules.d/99-oracle-asmdevices.rules

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29c9a269653c7cd04ea2edb0c71", NAME="ASM_DISK_2G_1", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29861258ebc95eded634a1f36fd", NAME="ASM_DISK_2G_2", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c296fd09d9ed213df1999820ca89", NAME="ASM_DISK_2G_3", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c2934ae12fd4af0dd366d85b42f5",  NAME="ASM_DISK_2G_4", WNER="grid", GROUP="asmadmin", MODE="0660"

 

#KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

#RESULT=="36000c2968513d97eb7cdd25bdbd10600",  NAME="ASM_DISK_1G_1", WNER="grid", GROUP="asmadmin", MODE="0660"

 

#KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

#RESULT=="36000c29682ba5ec7a4c8a189499a89bb",  NAME="ASM_DISK_1G_2", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c292aa4a4ebbb25d15d2084c01ca",  NAME="ASM_DISK_1G_3", WNER="grid", GROUP="asmadmin", MODE="0660"

 

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id",

RESULT=="36000c29a98ca1bd8a0012e99f992a19d",  NAME="ASM_DISK_1G_4", WNER="grid", GROUP="asmadmin", MODE="0660"

 

重新啟動udev

[root@ora32 dev]# /sbin/udevcontrol reload_rules

[root@ora32 dev]# /sbin/start_udev

Starting udev: [  OK  ]

 

查看一下dev下的設備,發現仍然存在/dev/ASM_DISK_1G_1/dev/ASM_DISK_1G_2

[root@ora32 dev]# ll /dev/ASM*

brw-rw---- 1 grid asmadmin 8,  17 May 13 16:15 /dev/ASM_DISK_1G_1

brw-rw---- 1 grid asmadmin 8,  97 May 13 16:15 /dev/ASM_DISK_1G_2

brw-rw---- 1 grid asmadmin 8, 113 May 13 16:38 /dev/ASM_DISK_1G_3

brw-rw---- 1 grid asmadmin 8, 129 May 13 16:37 /dev/ASM_DISK_1G_4

brw-rw---- 1 grid asmadmin 8,  33 May 13 16:38 /dev/ASM_DISK_2G_1

brw-rw---- 1 grid asmadmin 8,  49 May 13 16:38 /dev/ASM_DISK_2G_2

brw-rw---- 1 grid asmadmin 8,  65 May 13 16:38 /dev/ASM_DISK_2G_3

brw-rw---- 1 grid asmadmin 8,  81 May 13 16:37 /dev/ASM_DISK_2G_4

 

不知道是何原因。將數據庫實例和ASM實例都停掉后,重新啟動udev服務后發現dev下仍然存在/dev/ASM_DISK_1G_1/dev/ASM_DISK_1G_2,通過udevinfo查詢發現已經在udev不存在記錄。

 

[root@ora32 dev]# udevinfo -q path -n /dev/ASM_DISK_1G_1

no record for 'ASM_DISK_1G_1' in database

[root@ora32 dev]# udevinfo -q path -n /dev/ASM_DISK_1G_2

no record for 'ASM_DISK_1G_2' in database

[root@ora32 dev]#

[root@ora32 dev]# udevinfo -q path -n /dev/ASM_DISK_1G_3

/block/sdh/sdh2

 

 

在網上查閱了資料,并未找到解決/dev/ASM_DISK_1G_1/dev/ASM_DISK_1G_2為什么仍然存在的原因,繼續跟蹤這個問題。

向AI問一下細節

免責聲明:本站發布的內容(圖片、視頻和文字)以原創、轉載和分享為主,文章觀點不代表本網站立場,如果涉及侵權請聯系站長郵箱:is@yisu.com進行舉報,并提供相關證據,一經查實,將立刻刪除涉嫌侵權內容。

AI

湖口县| 岳西县| 明水县| 紫金县| 泸州市| 西充县| 象山县| 纳雍县| 东乌| 巩义市| 长治市| 石泉县| 福贡县| 土默特左旗| 永胜县| 门头沟区| 舞钢市| 攀枝花市| 乐都县| 屏南县| 从化市| 沂水县| 宝丰县| 五大连池市| 博乐市| 治多县| 楚雄市| 蒙山县| 南溪县| 白山市| 阿勒泰市| 会泽县| 保康县| 双鸭山市| 司法| 乐山市| 铁力市| 广宁县| 本溪| 镇赉县| 岳池县|