CentOS7.6啟用Ceph nautilus的dashboard失敗處理

[TOC]

1. 環境說明

ceph-deploy的nautilus版本安裝和luminous版本一樣，可以參考我以前的文章：
https://blog.51cto.com/ygqygq2/2161917

2. 啟用ceph dashboard失敗

mgr模塊開啟dashboard提示錯誤，雖然加上--force表面啟用了，但實際并未啟動dashboard服務，因為設置dashboard用戶命令提示不支持。

3. 問題排查過程

mgr節點查看mgr啟動日志

詳細日志：

2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  e: '/usr/bin/ceph-mgr'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  0: '/usr/bin/ceph-mgr'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  1: '-f'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  2: '--cluster'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  3: 'ceph'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  4: '--id'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  5: 'utyun-node2'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  6: '--setuser'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  7: 'ceph'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  8: '--setgroup'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  9: 'ceph'
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn respawning with exe /usr/bin/ceph-mgr
2019-11-08 12:44:23.673 7f7388cd9700  1 mgr respawn  exe_path /proc/self/exe
2019-11-08 12:44:23.737 7fb0b7715b80  0 ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable), process ceph-mgr, pid 158648
2019-11-08 12:44:23.738 7fb0b7715b80  0 pidfile_write: ignore empty --pid-file
2019-11-08 12:44:23.776 7fb0b7715b80  1 mgr[py] Loading python module 'ansible'
2019-11-08 12:44:23.834 7fb0b7715b80 -1 mgr[py] Module not found: 'ansible'
2019-11-08 12:44:23.834 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/ansible/__init__.py", line 5, in <module>
    from .module import Module
  File "/usr/share/ceph/mgr/ansible/module.py", line 10, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:23.835 7fb0b7715b80 -1 mgr[py] Class not found in module 'ansible'
2019-11-08 12:44:23.835 7fb0b7715b80 -1 mgr[py] Error loading module 'ansible': (2) No such file or directory
2019-11-08 12:44:23.835 7fb0b7715b80  1 mgr[py] Loading python module 'balancer'
2019-11-08 12:44:23.850 7fb0b7715b80  1 mgr[py] Loading python module 'crash'
2019-11-08 12:44:23.864 7fb0b7715b80  1 mgr[py] Loading python module 'dashboard'
2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Module not found: 'dashboard'
2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/dashboard/__init__.py", line 38, in <module>
    from .module import Module, StandbyModule
  File "/usr/share/ceph/mgr/dashboard/module.py", line 75, in <module>
    from .grafana import push_local_dashboards
  File "/usr/share/ceph/mgr/dashboard/grafana.py", line 7, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Class not found in module 'dashboard'
2019-11-08 12:44:24.133 7fb0b7715b80 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
2019-11-08 12:44:24.134 7fb0b7715b80  1 mgr[py] Loading python module 'deepsea'
2019-11-08 12:44:24.197 7fb0b7715b80 -1 mgr[py] Module not found: 'deepsea'
2019-11-08 12:44:24.197 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/deepsea/__init__.py", line 1, in <module>
    from .module import DeepSeaOrchestrator
  File "/usr/share/ceph/mgr/deepsea/module.py", line 13, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:24.198 7fb0b7715b80 -1 mgr[py] Class not found in module 'deepsea'
2019-11-08 12:44:24.198 7fb0b7715b80 -1 mgr[py] Error loading module 'deepsea': (2) No such file or directory
2019-11-08 12:44:24.198 7fb0b7715b80  1 mgr[py] Loading python module 'devicehealth'
2019-11-08 12:44:24.211 7fb0b7715b80  1 mgr[py] Loading python module 'influx'
2019-11-08 12:44:24.226 7fb0b7715b80  1 mgr[py] Loading python module 'insights'
2019-11-08 12:44:24.240 7fb0b7715b80  1 mgr[py] Loading python module 'iostat'
2019-11-08 12:44:24.254 7fb0b7715b80  1 mgr[py] Loading python module 'localpool'
2019-11-08 12:44:24.268 7fb0b7715b80  1 mgr[py] Loading python module 'orchestrator_cli'
2019-11-08 12:44:24.304 7fb0b7715b80  1 mgr[py] Loading python module 'pg_autoscaler'
2019-11-08 12:44:24.350 7fb0b7715b80  1 mgr[py] Loading python module 'progress'
2019-11-08 12:44:24.383 7fb0b7715b80  1 mgr[py] Loading python module 'prometheus'
2019-11-08 12:44:24.446 7fb0b7715b80  1 mgr[py] Loading python module 'rbd_support'
2019-11-08 12:44:24.523 7fb0b7715b80  1 mgr[py] Loading python module 'restful'
2019-11-08 12:44:24.674 7fb0b7715b80  1 mgr[py] Loading python module 'selftest'
2019-11-08 12:44:24.689 7fb0b7715b80  1 mgr[py] Loading python module 'status'
2019-11-08 12:44:24.714 7fb0b7715b80  1 mgr[py] Loading python module 'telegraf'
2019-11-08 12:44:24.737 7fb0b7715b80  1 mgr[py] Loading python module 'telemetry'
2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Module not found: 'telemetry'
2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Traceback (most recent call last):
  File "/usr/share/ceph/mgr/telemetry/__init__.py", line 1, in <module>
    from .module import Module
  File "/usr/share/ceph/mgr/telemetry/module.py", line 10, in <module>
    import requests
  File "/usr/lib/python2.7/site-packages/requests/__init__.py", line 58, in <module>
    from . import utils
  File "/usr/lib/python2.7/site-packages/requests/utils.py", line 32, in <module>
    from .exceptions import InvalidURL
  File "/usr/lib/python2.7/site-packages/requests/exceptions.py", line 10, in <module>
    from .packages.urllib3.exceptions import HTTPError as BaseHTTPError
  File "/usr/lib/python2.7/site-packages/requests/packages/__init__.py", line 95, in load_module
    raise ImportError("No module named '%s'" % (name,))
ImportError: No module named 'requests.packages.urllib3'

2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Class not found in module 'telemetry'
2019-11-08 12:44:24.784 7fb0b7715b80 -1 mgr[py] Error loading module 'telemetry': (2) No such file or directory
2019-11-08 12:44:24.784 7fb0b7715b80  1 mgr[py] Loading python module 'test_orchestrator'
2019-11-08 12:44:24.826 7fb0b7715b80  1 mgr[py] Loading python module 'volumes'
2019-11-08 12:44:24.940 7fb0b7715b80  1 mgr[py] Loading python module 'zabbix'
2019-11-08 12:44:24.957 7fb0b7715b80 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: ansible, dashboard, deepsea, telemetry
2019-11-08 12:44:24.961 7fb0a2b49700  0 ms_deliver_dispatch: unhandled message 0x5601d8ebaa00 mon_map magic: 0 v1 from mon.0 v2:172.16.138.26:3300/0

嘗試pip手動安裝模塊，發現不管裝什么插件都報錯，通過升級python解決了pip安裝問題：yum -y update python。

但pip安裝ansible模塊死活裝不上，并且啟動dashboard仍舊提示ImportError: No module named 'requests.packages.urllib3'

從日志上看，好像ceph mgr啟用模塊，原理上是python使用urllib3從網絡上下載安裝的，所以換個方向，先解決urllib3模塊問題。

pip卸載urllib3，改用yum方式安裝。

再次實時查看啟用dashboard模塊時的日志，
tail -f /var/log/ceph/ceph-mgr.utyun-node2.log

終于如愿以償，dashboard啟用成功，并沒有任何報錯。

[root@utyun-node1 external-ceph]# ceph mgr module ls |more
{
    "enabled_modules": [
        "dashboard",
        "iostat",
        "pg_autoscaler",
        "restful"
    ],

4. 小結

以下總結ceph nautilus開啟dashboard的步驟：

mgr節點操作：

yum -y install ceph-mgr-dashboard
pip uninstall urllib3
yum install python-urllib3 -y

ceph管理節點操作：

ceph mgr module enable dashboard
ceph config set mgr mgr/dashboard/server_addr 0.0.0.0
ceph config set mgr mgr/dashboard/server_port 7000
ceph config set mgr mgr/dashboard/ssl false
ceph dashboard ac-user-create admin password administrator

參考資料：
[1] https://docs.ceph.com/docs/nautilus/mgr/dashboard/