AIX 系統故障之--擴展文件系統故障

AIX 系統故障之--擴展文件系統故障

系統環境：

  操作系統： AIX5300-08

  Oracle：  Oracle 10gR2（64）

故障現象：

     smit chfs 擴展 “ /" 時，提示找不到”device “，不能擴展；在rootvg 擴展pv 后，后來pv 被破壞，導致ODM出錯

1、查看rootvg

[root@aix216 /]#lsvg -p rootvg

rootvg:

PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION

hdisk0            active            546         161         107..00..00..12..42

0516-304 : Unable to find device id 0000c908116d1325 in the Device

       Configuration Database.

0000c908116d1325  missing           320         320         64..64..64..64..64

[root@aix216 /]#

----嘗試解決

[root@aix216 /]#redefinevg -d hdisk0 rootvg

[root@aix216 /]#lsvg -p rootvg

rootvg:

PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION

hdisk0            active            546         161         107..00..00..12..42

0516-304 : Unable to find device id 0000c908116d1325 in the Device

       Configuration Database.

0000c908116d1325  missing           320         320         64..64..64..64..64

[root@aix216 /]#lspv

hdisk0          0027713e1b5646e5                    rootvg          active

[root@aix216 /]#reducevg

0516-606 reducevg: Volume group name not entered.

Usage: reducevg [-d] [-f] VGname PVname...

Reduces volume group size by removing a physical volume.

2、從rootvg 中刪除hdisk0

------ reducevg 刪除pv 后，hd5 被刪除

[root@aix216 /]#reducevg rootvg 0027713e1b5646e5

0516-016 ldeletepv: Cannot delete physical volume with allocated

       partitions. Use either migratepv to move the partitions or

       reducevg with the -d option to delete the partitions.

0516-884 reducevg: Unable to remove physical volume 0027713e1b5646e5.

[root@aix216 /]#reducevg rootvg 0027713e1b5646e5 -d

0516-306 getlvodm: Unable to find physical volume -d in the Device

       Configuration Database.

0516-016 ldeletepv: Cannot delete physical volume with allocated

       partitions. Use either migratepv to move the partitions or

       reducevg with the -d option to delete the partitions.

0516-884 reducevg: Unable to remove physical volume 0027713e1b5646e5.

0516-306 getlvodm: Unable to find physical volume -d in the Device

       Configuration Database.

0516-884 reducevg: Unable to remove physical volume -d.

[root@aix216 /]#reducevg -d rootvg 0027713e1b5646e5

0516-914 rmlv: Warning, all data belonging to logical volume

       hd5 on physical volume 0027713e1b5646e5 will be destroyed.

rmlv: Do you wish to continue? y(es) n(o)?

y

0516-1246 rmlv: If hd5 is the boot logical volume, please run 'chpv -c <diskname>'

       as root user to clear the boot record and avoid a potential boot

       off an old boot p_w_picpath that may reside on the disk from which this

       logical volume is moved/removed.

rmlv: Logical volume hd5 is removed.

0516-1734 rmlv: Warning, savebase failed.  Please manually run 'savebase' before rebooting.

0516-914 rmlv: Warning, all data belonging to logical volume

       fwdump on physical volume 0027713e1b5646e5 will be destroyed.

rmlv: Do you wish to continue? y(es) n(o)?

yes

0516-1008 rmlv: Logical volume fwdump must be closed.  If the logical volume

       contains a filesystem, the umount command will close the LV device.

0516-1734 rmlv: Warning, savebase failed.  Please manually run 'savebase' before rebooting.

0516-884 reducevg: Unable to remove physical volume 0027713e1b5646e5.

0516-1734 reducevg: Warning, savebase failed.  Please manually run 'savebase' before rebooting.

[root@aix216 /]#savebase

[root@aix216 /]#

[root@aix216 /tmp]#lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd6                 paging     32      32      1    open/syncd    N/A

hd8                 jfs2log    1       1       1    open/syncd    N/A

hd4                 jfs2       136     136     1    open/syncd    /

hd2                 jfs2       24      24      1    open/syncd    /usr

hd9var              jfs2       12      12      1    open/syncd    /var

hd3                 jfs2       16      16      1    open/syncd    /tmp

hd1                 jfs2       16      16      1    open/syncd    /home

hd10opt             jfs2       24      24      1    open/syncd    /opt

fwdump              jfs2       1       1       1    open/syncd    /var/adm/ras/platform

fs001               jfs2       120     120     1    open/syncd    /u01

loglv00             jfslog     1       1       1    open/syncd    N/A

lv00                jfs        1       1       1    open/syncd    /var/adm/csd

3、重新建立 hd5

重建hd5 ，出錯

[root@aix216 /tmp]#mklv -y hd5 -t boot -a e rootvg 1

mklv -y hd5 -t boot -a e rootvg 1

0516-304 lquerypv: Unable to find device id 0000c908116d13250000000000000000 in the Device

       Configuration Database.

0516-634 mklv: /tmp directory does not have enough space,

       delete some files and try again.

0516-1734 mklv: Warning, savebase failed.  Please manually run 'savebase' before rebooting.

[root@aix216 /tmp]#bosboot -ad /dev/ipldevice

0516-306 lslv: Unable to find  hd5 in the Device

       Configuration Database.

0301-168 bosboot: The current boot logical volume, /dev/hd5,

       does not exist on /dev/hdisk0.

----- 提示：/tmp空間不足，實際/tmp空間很大，嘗試刪除tmp 下所有的文件

[root@aix216 /tmp]#df -m

Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on

/dev/hd4        7168.00   6032.57   16%     4899     1% /

/dev/hd2        3072.00   1207.03   61%    46571    15% /usr

/dev/hd9var     1536.00    696.15   55%     1170     1% /var

/dev/hd3        2048.00   2039.82    1%       25     1% /tmp

/dev/fwdump      128.00    127.66    1%        4     1% /var/adm/ras/platform

/dev/hd1        2048.00   2003.45    3%      233     1% /home

/proc                 -         -    -         -     -  /proc

/dev/hd10opt    1024.00    749.93   27%     4799     3% /opt

/dev/lv00        128.00    123.94    4%       18     1% /var/adm/csd

/dev/fs001     15360.00  15357.19    1%        7     1% /u01

[root@aix216 /tmp]# rm -rf /tmp/*        

4、再次刪除rootvg 下的 hdisk0

[root@aix216 /tmp]#lsvg -p rootvg

rootvg:

PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION

hdisk0            active            546         162         108..00..00..12..42

0516-304 : Unable to find device id 0000c908116d1325 in the Device

       Configuration Database.

0000c908116d1325  missing           320         320         64..64..64..64..64

[root@aix216 /tmp]#reducevg -d rootvg "0000c908116d1325"

0516-304 putlvodm: Unable to find device id 0000c908116d13250000000000000000 in the Device

       Configuration Database.

0516-896 reducevg: Warning, cannot remove physical volume 0000c908116d1325 from

       Device Configuration Database.

0516-1734 reducevg: Warning, savebase failed.  Please manually run 'savebase' before rebooting.

[root@aix216 /tmp]#synclvodm -v rootvg

synclvodm: Physical volume data updated.

synclvodm: Logical volume hd6 updated.

synclvodm: Logical volume hd8 updated.

synclvodm: Logical volume hd4 updated.

synclvodm: Logical volume hd2 updated.

synclvodm: Logical volume hd9var updated.

synclvodm: Logical volume hd3 updated.

synclvodm: Logical volume hd1 updated.

synclvodm: Logical volume hd10opt updated.

synclvodm: Logical volume fwdump updated.

synclvodm: Logical volume fs001 updated.

synclvodm: Logical volume loglv00 updated.

synclvodm: Logical volume lv00 updated.

[root@aix216 /tmp]#lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd6                 paging     32      32      1    open/syncd    N/A

hd8                 jfs2log    1       1       1    open/syncd    N/A

hd4                 jfs2       136     136     1    open/syncd    /

hd2                 jfs2       24      24      1    open/syncd    /usr

hd9var              jfs2       12      12      1    open/syncd    /var

hd3                 jfs2       16      16      1    open/syncd    /tmp

hd1                 jfs2       16      16      1    open/syncd    /home

hd10opt             jfs2       24      24      1    open/syncd    /opt

fwdump              jfs2       1       1       1    open/syncd    /var/adm/ras/platform

fs001               jfs2       120     120     1    open/syncd    /u01

loglv00             jfslog     1       1       1    open/syncd    N/A

lv00                jfs        1       1       1    open/syncd    /var/adm/csd

5、重新建立 hd5

繼續重建hd5 ，重建成功

[root@aix216 /tmp]#mklv -y hd5 -t boot -a e rootvg 1

mklv -y hd5 -t boot -a e rootvg 1

hd5

[root@aix216 /tmp]#bosboot -ad /dev/ipldevice

bosboot: Boot p_w_picpath is 37908 512 byte blocks.

[root@aix216 /tmp]#

6、故障解決

查看rootvg pv ，問題解決

[root@aix216 /tmp]#lsvg -p rootvg

rootvg:

PV_NAME           PV STATE          TOTAL PPs   FREE PPs    FREE DISTRIBUTION

hdisk0            active            546         161         107..00..00..12..42

[root@aix216 /tmp]#

[root@aix216 /tmp]#lsvg -l rootvg

rootvg:

LV NAME             TYPE       LPs     PPs     PVs  LV STATE      MOUNT POINT

hd5                 boot       1       1       1    closed/syncd  N/A

hd6                 paging     32      32      1    open/syncd    N/A

hd8                 jfs2log    1       1       1    open/syncd    N/A

hd4                 jfs2       136     136     1    open/syncd    /

hd2                 jfs2       24      24      1    open/syncd    /usr

hd9var              jfs2       12      12      1    open/syncd    /var

hd3                 jfs2       16      16      1    open/syncd    /tmp

hd1                 jfs2       16      16      1    open/syncd    /home

hd10opt             jfs2       24      24      1    open/syncd    /opt

fwdump              jfs2       1       1       1    open/syncd    /var/adm/ras/platform

fs001               jfs2       120     120     1    open/syncd    /u01

loglv00             jfslog     1       1       1    open/syncd    N/A

lv00                jfs        1       1       1    open/syncd    /var/adm/csd

[root@aix216 /tmp]#