UMOUNT NAS報錯No space left on device和was not found in /proc/mounts

UMOUNT NAS報錯No space left on device和was not found in /proc/mounts

NAS_IP:/sysidc 表示NAS的存儲設備。

[root@mvxl0280 /]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root  22G   20G  275M  99% /

tmpfs                 7.8G     0  7.8G   0% /dev/shm

/dev/sda1             485M   37M  423M   9% /boot

/dev/mapper/vg_01-lv_01  98G   72G   21G  78% /eam_app

NAS_IP:/sysidc  3.0T  2.2T  743G  76% /mnt

## mount命令不加參數顯示的結果和/etc/mtab文件記錄內容一樣。

[root@mvxl0280 /]# mount

/dev/mapper/VolGroup-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw)

/dev/sda1 on /boot type ext4 (rw)

/dev/mapper/vg_01-lv_01 on /eam_app type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

NAS_IP:/sysidc  on /mnt type nfs (rw,addr=NAS_IP)

[root@mvxl0280 /]# cat /proc/mounts

rootfs / rootfs rw 0 0

proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0

devtmpfs /dev devtmpfs rw,nosuid,relatime,size=8157104k,nr_inodes=2039276,mode=755 0 0

devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0

tmpfs /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0

/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,barrier=1,data=ordered 0 0

/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0

/dev/sda1 /boot ext4 rw,relatime,barrier=1,data=ordered 0 0

/dev/mapper/vg_01-lv_01 /eam_app ext4 rw,relatime,barrier=1,data=ordered 0 0

none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0

/etc/auto.misc /misc autofs rw,relatime,fd=7,pgrp=2080,timeout=300,minproto=5,maxproto=5,indirect 0 0

-hosts /net autofs rw,relatime,fd=13,pgrp=2080,timeout=300,minproto=5,maxproto=5,indirect 0 0

NAS_IP:/sysidc /mnt nfs rw,relatime,vers=3,rsize=32768,wsize=32768,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.16.17.171,mountvers=3,mountport=1234,mountproto=udp,local_lock=none,addr=NAS_IP 0 0

umount NAS存儲提示【No space left on device】

[root@mvxl0280 ~]# umount /mnt

can't create lock file /etc/mtab~9811: No space left on device (use -n flag to override)

/mnt was not found in /proc/mounts

NAS空間突然變小，從3.0T變成22G，其實此時看到的/mnt雖然掛載點是NAS“NAS_IP:/sysidc”，但其實是“/”目錄的掛載點。

[root@mvxl0280 ~]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root  22G   20G  346M  99% /

tmpfs                 7.8G     0  7.8G   0% /dev/shm

/dev/sda1             485M   37M  423M   9% /boot

/dev/mapper/vg_01-lv_01  98G   72G   21G  78% /eam_app

NAS_IP:/sysidc   22G   20G  346M  99% /mnt

其實umount執行之后，NAS存儲實際已經成功umount，通過查看/mnt目錄就知道已經是空的。

[root@mvxl0280 ~]# ls -l  /mnt

total 0

貌似是沒有卸載成功，但是此時再執行一次帶【-f】的umount，依然報錯。

[root@mvxl0280 ~]# umount -f /mnt

/mnt was not found in /proc/mounts

/mnt was not found in /proc/mounts

重新mount NAS存儲也提示【No space left on device】

[root@mvxl0280 /]# mount  NAS_IP:/sysidc   /mnt

can't create lock file /etc/mtab~10608: No space left on device (use -n flag to override)

網上說可以用umount的【-l】參數，其實也沒有效果。

[root@mvxl0280 /]# umount -l /mnt/

can't create lock file /etc/mtab~10512: No space left on device (use -n flag to override)

can't create lock file /etc/mtab~10513: No space left on device (use -n flag to override)

但是mount可以查看到NAS存儲“NAS_IP:/sysidc”已經掛載在/mnt下。

[root@mvxl0280 ~]# mount

/dev/mapper/VolGroup-lv_root on / type ext4 (rw)

proc on /proc type proc (rw)

sysfs on /sys type sysfs (rw)

devpts on /dev/pts type devpts (rw,gid=5,mode=620)

tmpfs on /dev/shm type tmpfs (rw)

/dev/sda1 on /boot type ext4 (rw)

/dev/mapper/vg_01-lv_01 on /eam_app type ext4 (rw)

none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

NAS_IP:/sysidc on /mnt type nfs (rw,addr=NAS_IP)

而/proc/mounts【/proc/self/mounts的軟連接】卻看不到掛載點信息，因此報錯【/mnt was not found in /proc/mounts】。

[root@mvxl0280 ~]# ls -l /proc/mounts

lrwxrwxrwx 1 root root 11 Feb  5 09:06 /proc/mounts -> self/mounts

[root@mvxl0280 ~]# file /proc/mounts

/proc/mounts: symbolic link to `self/mounts'

[root@mvxl0280 ~]# ls -l /proc/self

lrwxrwxrwx 1 root root 64 Feb  5 08:57 /proc/self -> 9879

[root@mvxl0280 ~]# ls -l /proc/self/mounts

-r--r--r-- 1 root root 0 Feb  5 09:07 /proc/self/mounts

[root@mvxl0280 ~]# more /proc/self/mounts

rootfs / rootfs rw 0 0

proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0

sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0

devtmpfs /dev devtmpfs rw,nosuid,relatime,size=8157104k,nr_inodes=2039276,mode=755 0 0

devpts /dev/pts devpts rw,relatime,gid=5,mode=620,ptmxmode=000 0 0

tmpfs /dev/shm tmpfs rw,nosuid,nodev,relatime 0 0

/dev/mapper/VolGroup-lv_root / ext4 rw,relatime,barrier=1,data=ordered 0 0

/proc/bus/usb /proc/bus/usb usbfs rw,relatime 0 0

/dev/sda1 /boot ext4 rw,relatime,barrier=1,data=ordered 0 0

/dev/mapper/vg_01-lv_01 /eam_app ext4 rw,relatime,barrier=1,data=ordered 0 0

none /proc/sys/fs/binfmt_misc binfmt_misc rw,relatime 0 0

sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0

/etc/auto.misc /misc autofs rw,relatime,fd=7,pgrp=2080,timeout=300,minproto=5,maxproto=5,indirect 0 0

-hosts /net autofs rw,relatime,fd=13,pgrp=2080,timeout=300,minproto=5,maxproto=5,indirect 0 0

經過對比分析，發現是“/”根目錄空間不足導致，而使用空間最多是一個/home/wasadmin/deploy的程序備份目錄，在跟開發人員確認之后，這個目錄下的程序文件是定期備份，但是沒有部署相關的定期清理腳本，而導致空間被用完。

[root@mvxl0280 home]# du -sh *

60K     monitor

13G     wasadmin

/home/wasadmin是用戶的home目錄，使用13G空間，而該目錄下只有2個常規目錄。其中client_ffdc這個目錄才204K。那么deploy占用了剩余的其他12G多的空間。

[root@mvxl0280 home]# cd  wasadmin

[root@mvxl0280 wasadmin]# ls

client_ffdc  deploy

[root@mvxl0280 wasadmin]# du -sh client_ffdc/

204K    client_ffdc/

在刪除/home/wasadmin/deploy目錄下的部分備份程序之后，“/”目錄的空間已經足夠，再次執行umount NAS的命令，可以成功執行。

[wasadmin@mvxl0280 ~]$ df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root  22G  8.9G   12G  45% /

tmpfs                 7.8G     0  7.8G   0% /dev/shm

/dev/sda1             485M   37M  423M   9% /boot

/dev/mapper/vg_01-lv_01  98G   68G   26G  73% /eam_app

NAS_IP:/sysidc  3.0T  2.2T  743G  76% /mnt

[root@mvxl0280 wasadmin]# umount /mnt

[root@mvxl0280 wasadmin]# df -h

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/VolGroup-lv_root 22G  8.9G   12G  45% /

tmpfs                 7.8G     0  7.8G   0% /dev/shm

/dev/sda1             485M   37M  423M   9% /boot

/dev/mapper/vg_01-lv_01 98G   68G   26G  73% /eam_app